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Abstract. Let Ak be the set of permutations in the symmetric group &k with prefix 12. 
This paper concerns the enumeration of involutions which avoid the set of patterns Ak ■ We 
present a bijection between symmetric Schroder paths of length 2n and involutions of length 
n + 1 avoiding A^. Statistics such as the number of right-to-left maxima and fixed points 
of the involution correspond to the number of steps in the symmetric Schroder path of a 
particular type. For each fc > 3 we determine the generating function for the number of 
involutions avoiding the subsequences in Ak , according to length, first entry and number of 
fixed points. 



1. Introduction 

Let &n denote the set of permutations of [n] = {1, . . . , n}, written in one-line notation. For 
two permutations vr G ©„ and r G &k, an occurrence of r in vr is a subsequence 1 < ii < ^2 < 
■ ■ ■ < ik ^ n such that (tTj^, . . . ,vrjj.) is order-isomorphic to r; in such a context r is usuahy 
called a pattern. We say that vr avoids r, or is t- avoiding, if there is no occurrence of r in 
vr. A natural generalization of single pattern avoidance is subset avoidance, that is, we say 
that vr G S„ avoids a subset T C if vr avoids all t £ T. The set of all r-avoiding (resp. 
T-avoiding) permutations of length n is denoted Sri(r) (resp. (3„(T)). 

Several authors have considered the case of general k in which T enjoys various algebraic 
properties. Barcucci et al. [2j treat the case of permutations avoiding the collection of per- 
mutations in 6k that have suffix {k — l)k. Adin and Roichman [1 look at the case where T 
is a Kazhdan-Lusztig cell of &k, or, equivalently, a Knuth equivalence class (see |12| Vol. 2, 
Ch. Al]). Mansour and Vainshtein [10] consider the situation where T is a maximal parabolic 
subgroup of 6fc. 

In the current paper an analogous result is established for pattern-avoiding involutions. We 
say vr is an involution whenever tt-,^. = i for all i £ [n]. Let 2n denote the set of involutions 
of [n]. The set of all r-avoiding (resp. T-avoiding) involutions of length n is denoted Tn{T) 
(resp. ln{T)). 

Simion and Schmidt fTP considered the first cases of pattern- avoiding involutions, which was 
continued in Gouyou-Beauchamps [5j and Gessel [6j for increasing patterns, and subsequently 
in Guibert's Ph.D. thesis [7]. This paper concerns the enumeration of involutions which avoid 
the class of permutations in &k with prefix 12, that is, 

Ak = {vrivr2 . . . vrfc G ©A; I vri = 1, vr2 = 2}. 
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We denote by In the cardinality of the set 2„. We say that i is a fixed point of a permutation 
vr if TTj = i. Define Jn{p) to be the polynomial Yl^=o -^n-jP^ , where In-j is the number of 
involutions in "Xyi with j fixed points. For example Jsi^p) — 3p -t- p"^. It is not hard to see 
that the polynomials Jn{p) satisfy the recurrence relation Jn{p) = pJn-i{p) + {n — l)Jn-2{p), 
n > 2, with the initial conditions Jo{p) = 1 and Ji{p) = p. The exponential generating 
function for the sequence {Jn{p)}n>o is given by e^^^^ 

The main result of this paper can be formulated as follows. 

Theorem 1.1. Let k > 2. The generating function for the number of Ak-avoiding involu- 
tions of length n is given by 

^np#fixed points{n) _ 

2I1 -^Av)^^ 2^ ( ^ ~^ ^^^^ ~ ^''^^ - 2a; - p)uq{x) \ Jk-2{p) 



X 



X + p- {x^{k - 3) - px^{k - 1) + x + p)uo{x) I Jk-sip) 



where uo{x) = 1/^/1 - 2{k - l)x^ + {k - 3)^x^. 

The proof is given in Section 3. Theorem 1 1 . 1 1 with A; = 3 and p = 1 shows the generating func- 
tion for the number 123-avoiding involutions of length n to be i+yr TS^ ~ ^n>o ([n/2j)^" 
(see [H]). Also, Theorem with k = 3 and p = gives the number of 123 -avo iding invo- 



1.1 



with k 



lutions of length 2n without fixed points to be ^ ( ^^^2\ ) • Moreover, Theorem 
and p = 1 gives the generating function for the number {1234, 1243}-avoiding involutions of 
length n to be + ^^\j^^E^- Section 2 we present a bijection between symmetric 
Schroder paths of length n — 1 and {1234, 1243}-avoiding involutions of length n, thereby 
providing a combinatorial proof of the above result with A; = 4. 

2. Symmetric Schroder paths and {1234, 1243}-avoiding involutions 

A Schroder path of length 2n is a lattice path from (0,0) to (2n, 0) consisting of double 
horizontal steps h = (2,0), up steps u = (1,1) and down steps d = (1,-1) that never goes 
below the x-axis. The set of all Schroder paths of length 2n is enumerated by the n-th 
Schroder number. Kremer [51 Corollary 9] showed that S„(1243, 2143) is also enumerated by 
the n-th Schroder number. 

A Schroder path of length 2n is called symmetric if it is symmetric about the line x = n. Let 
Sh„ be the collection of all such paths. In this section we give a bijection between symmetric 
Schroder paths of length 2n and the class of involutions in Sn+i that avoid the patterns 
{1234,1243}. 

We will now describe a map (j) : Sh„ X„-(.i(1234, 1243). Alongside this description will be 
an example of the map acting on the path p = uhuduudhuddudhd G Shg. Several points may 
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at first appear extraneous, however these will be required in the proof of the Theorem which 
follows. 




Given p G Sh„, let p' be the word of length n + 1 obtained from p by first appending a 
horizontal step h to the end, then replacing all occurrences of ud in p by r, and finally 
deleting all remaining d's. This is equivalent to projecting the steps of the path onto the 
diagonal, and replacing any u's that are followed by a d by r. 




Prom the diagram the example path is p' = uhrurhrrhh. In general, p' = pi...pn+i £ 
{u, r, h}"+^. Now form the three sets Ah, Ar, A^J, according to the rule i ^ Ax\ipi = x. 




In the example, A = {2,6,9,10}, A, = {3,5,7,8} and A^ = {1,4}. 

Starting with the largest entry in Ah, replace all h's in p' (from left to right) with the entries 
from Ah so that this sequence is decreasing. This is equivalent to forming a sequence of 
transpositions (or a fixed point), the first of which will have as first entry the index of the 
first h in p' and whose second entry is the index of the last h in p'. The second transposition 
(or fixed point) will have as first entry the index of the second h in p' and whose second entry 
is the index of the second last h in p', and so forth. 



Since Ah 



= {2, 6, 9, 10} in the example, we have the transpositions (2, 10) and (6, 9). 
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Do likewise for the sets and A^. The label of a particular h, r or u in is the value that 
replaces it. Call the resulting permutation tt = 4'{p), i.e. the labels of p' read from left to 
right. 

The label of the first h in p' above is 10 and the label of the third h in p' above is 6. 
In the example, since A^ = {2,6,9,10}, Ar = {3,5,7,8} and A^ = {1,4}, tt = ^(p) is 
the permutation (involution) with cycles (2, 10), (6, 9), (3, 8), (5, 7) and (1,4). Thus we have 
0(p) = (4,10,8,1,7,9,5,3,6,2). 




Another way to see this construction is in terms of layers of right-to-left maxima. An element 
TTi of a permutation tt is called a right-to-left maximum if it is greater than all elements that 
follow it, i.e. TTi > TTj for all j > i. We define successively the r-right-to-lcft maxima for a 
permutation tt G ©n- Let tt^^^ be the word consisting of all elements of tt. For r > 1, the 
right-to-left maxima of TT^'-) are called r -right-to-left maxima of tt. Let tt^^"*"^) be the subword 
obtained from tt^^^ by removing all r-right-to-left maxima. For example, the permutation 
TT = 674583912 G Sg has the 1-right-to-left maxima 9 and 2; the 2-right-to-lcft maxima 8, 3 
and 1; the 3-right-to-left maxima 7 and 5; and the 4-right-to-left maxima 6 and 4. Note that 
the r-right-to-left maxima of tt form a decreasing subsequence for each r. 

Let TT be the unique permutation (in fact, it will be an involution) in ©n+i with 1 right-to-left 
maxima A^, 2 right-to-left maxima Ar and 3 right-to-left maxima A^. If A^ = {xi, . . . , Xq,}, 
Ar = {yi, . ■ • , y/s} and A^, = {zi, . . . , z^} then the cycles of tt are 

(Xi Xa) {X2 Xa-l) ■■■ {yi 2//?) (?/2 y/3-l) " " " {Zl Z-y) {Z2 ^^-l) ■ ■ • . 



We point out that if 7 = 2m + 1 then (xm+i) will be a fixed point. Consequently there will 
be at most three fixed points in the resulting involution. 
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TT = (10,5,8,7,2,6,4,3,9,1) G Tio(1234, 1243). 



The sets of 1, 2 and 3 right-to-lcft maxima are Ay, = {1,9,10}, = {3,4,6,7,8} and 
Au = {2,5}, respectively. This gives p' = hurrurrrhh. After removing the final h, we have 




Beginning with the last step (at position 9), we push this down so that it is symmetric with 
the first entry. We then move the second step of p' down to meet the path. It is u so there 
must be a d inserted at the opposite end so the path is symmetric. 




Next we move the r at position 3 down to touch the evolving path, and move the r at position 
8 down to meet the path above the d step. 




The u at position 4 is moved next but we must insert a d step between positions 6 and 7 to 
ensure the path is symmetric. 
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Finally, move the remaining pieces down, inserting d's where appropriate. 




Thus we have p = (j) ^(tt) = huududuuddududdh. 

Theorem 2.1. The map (f) : Sh„ — Xn+i(1234, 1243) is a bijection. 

Proof. We first show that for any p G Sh„, the corresponding tt = (^(p) G Xn+i(1234, 1243). 

Let p G Shji and p' = pi- ■ ■ pk be the corresponding word on the alphabet {u, r, h}. Suppose 
that = {ii, . . . , if }. Then it is clear that iTi- = i^+i-j for all 1 < j < ^. The same is true 
for the sets and so tt is an involution. 

Prom the labelling scheme above, the resulting permutation tt has, at most, three levels of 
right-to-left maxima. It is therefore 1234 avoiding. To show that tt is 1243-avoiding, suppose 
TT contains a 1243 pattern vrjTrjTrfcvr^ , where VTj < VTj < vr^ < vr^ and i < j < k < i. The 4 and 
3 of the pattern (vrfc and tt^) must be in the 1 right-to-left maxima ^h- Similarly, the 2 (resp. 
1) of the pattern must be in the 2 (resp. 3) right-to-left maxima Ar (resp. ^u)- Then TrjTTfcTr^ 
is a 132-pattern with ttj G A^ and vrfe, vr^ G A^. This is not possible for the following reason: 
Given r in p' with label ri, let hi,h2, . . . be the sequence of labels of h's to the right of r. 
Then hi > ri > /i2, /i3, • • .. This statement is easily seen by removing all u's and the suffix 
h from p' and relabelling. (This relabelling always gives a monotone decreasing sequence.) 
The fact that the label of the first h after r is greater than the label of the r is due to the 
appended h. 

We now show how to construct the unique path p corresponding to tt G X„+i(1234, 1243). 
For such a permutation, let A^, A^ and be the 1, 2 and 3 right-to- left maxima of tt, 
respectively. Insert h at position z of p' if i G A^ and do likewise for the sets Ay and A^. 
Remove the suffix h from p' (it is a suffix since (n + 1) is one of the 1 right-to- left maxima). 
From right to left in p' , insert a d where there is a corresponding u and finish by replacing all 
occurrences of r with ud. (As was done in the example that preceded the Theorem.) We note 
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that for each there wiU be several Schroder paths to which it may correspond, however 
only one of these is symmetric. □ 

From the construction, we also have the following statistics of {1234, 1243}-avoiding involu- 
tions: 

Corollary 2.2. Let p G Sh„ with h steps h, r steps ud, and u steps u that are not directly 
followed by ad step. Let vr = (j){p) £ Tn(1234, 1243). 

(1) The number of right-to-left maxima of n is /i + 1. 

(2) The number of 2 right-to-left maxima of tt is r. 

(2) The number of 3 right-to-left maxima of vr is u. 

(3) The number of fixed points of n is ((1 + h) mod 2) + (r mod 2) + {u mod 2). 
Open Problem 2.3. What statistic on tt = (t){p) corresponds to the height of the path p? 

3. Proof of Theorem 11.11 



To present the proof of Theorem |1.1[ we must first consider the enumeration problem for the 
number .F^-avoiding involutions according to length and number of fixed points, where J-^ is 
the set of all permutations o" G with cti = \. 

3.1. Involutions avoiding Tt- In this subsection we present an explicit formula for the 
number of involutions that avoid all the patterns in Tk- To do so we require some new 
notation. Define fk{n) to be the number of involutions vr G Zn{J'k)- Given t G [n], we also 
define 

fk]mi'n; t) = #{7r G Zn{J^k) \t^i ='t and vr contains m fixed points}. 

Let fk{n;t) = fk{n,p;t) and /^(n) = fk{n,p) be the polynomials Ylm=o fk;ni{n;t)p"' and 
SILi fkiiT-'^t), respectively. We denote by Fk{x,p) the generating function for the sequence 
fk{n,p), that is Fk{x,p) = T.n>o fk{n,p)x'' ■ 

Theorem 3.1. We have 

Fk{x,p) = Jj{P)x^ + 2 ((^ - ^)Jk-2{p)x + Jk-l{p))- 

Moreover, the number of involutions of length k + 2n (resp. k + 2n — 1) that avoid all the 
patterns in J^k is given by {k — l)'"'^'^Lk-2 (resp. {k — l)'^Lk~i), for all n > 0. 

Proof. Let vr G S„ be a permutation that avoids all patterns in J^k- We have vri > n -\- 2 — k. 
Thus vr G Zn{^k) with vri=t>n-|-2 — A;if and only if vr2 . . . T^t-iT^t+i . . . vr„ is an involution 
on the numbers 2, . . . , t — 1, t + 1, . . . , n that avoids all the patterns in Tk- Hence, fk{n] j) = 
fk{n — 2) for all j = n + 2 — /c, n + 3 — fe, . . . , n, and fk{n, j) = for all j = 1,2, . . . ,n + l — k, 
where n > k. Thus, for n > k, 



fk{n) = {k-l)fk{n-2). 
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Using the initial conditions fk{j) = Jjip), j = 1,2, . . . ,k — 1, we find that /^(A; + 2j) = 
{k — l)-^"^^ Jfc_i(p) and fk{k + 2j — 1) = (k— 1)-' Jfc-2(p) for all J > 0. Rewriting these formulas 
in terms of generating functions we obtain 

Fk{x,p) = J2 JjiP)^^ + i_^L._i\ 2 ((^ ~ l)Jk-2{p)x + Jk-l{p)), 
j=Q ^ >^ 

as claimed. □ 

3.2. Involutions avoiding Ak- In this subsection we prove Theorem In order to do 
this, define gk{n) to be the number of involutions vr G Zn[Ak) and given ti,t2, ■ ■ ■ ,tm G N, 
we also define 

gk{n; ti,t2, . . ■ , tm) = #{vri . . . 7r„ G ln[Ak) \ tti . . .i^m = h ■ ■ ■ tm}- 
Lemma 3.2. Let k>3. For all 3 < t < n + 1 - k, 

t-2 

9k{n;t) = {k-2)gk{n-2;t-l) + ^gk{n-2;j), 

i=i 

with the initial conditions gk{n;l) = fk-i{n — 1), gk{n;2) = fk-i{n — 2), and gk{n;t) = 
gk{n — 2) for all t = n + 2 — k,n + 3 — k, . . . ,n. 

Proof. Let vr be any involution of length n that avoids all patterns in Ak with tti = t. Now 
let us consider all possible values of t. If t = 1 then vr S Tn{Ak) if and only if (vr2 — l)(vr3 — 

1) . . . (vr„ - 1) e In-i{J^k-i)- lit = 2 then vr G T„(A) if and only if (vrs - 2)(vr4 - 2) . . . (vr„ - 

2) G Zn-2{J^k~i)- Now assume that 3<t<n + l — k, then from the above definitions 

gk{n; t) = gk{n; t, 1) + . . . + gk{n; t,t-l)+ gk{n; t,t + l)-\ h gk{n; t, n). 

But any involution vr satisfying vri<vr2<'/i + 2 — A; contains a pattern from the set Ak (see 
the subsequence of the letters vri, vr2, n + 3 — fc, n + 4 — A;, . . . , n in vr). Thus gk{n; t,r) = for 
allt<r<n + 2 — A; and so 

gk{n; t) = gk{n; t,l) + ... + gk{n; t,t-l)+ gk{n; t,n + 3-k)^ h gk{n; t, n). 

Also, if vr is an involution in X„ with tti = t and vr2 = r>n + 3 — A;, then the entry r does 
not appear in any occurrence of r G Ak in vr. Thus, there exists a bijection between the set 
of involutions vr G Xn{Ak) with vri = t and vr2 = r>n + 3 — A; and the set of involutions 
vr' G In-2{Ak) with vr' = f — 1. Therefore gk{n; t, r) = gk{n — 2;t — \) which gives 

gk{n; t) = gk{n; t,l) + . . . + gk{n; t,t - I) + {k - 2)gk{n -2;t-l). 

Also, if vr is an involution in X„ with vri = t, vr2 = r < t and if ta2 ■ ■ - Qk is an occurrence 
of a pattern from the set Ak in vr, then ra2 ■ ■ - ak is an occurrence of a pattern from the 
set Ak in vr. Thus, there exists a bijection between the set of involutions vr G In{Ak) with 
vri = t and vr2 = r < t and the set of involutions vr' G In-2{Ak) with vr^ = r — 1. Therefore 
gk{n;t,r) = gk{n - 2; r - 1) which gives 

t-2 

gk{n;t) = {k - 2)gk{n - 2;t - I) + ^gk{n - 2; j), 

i=i 

as required. Finally, if vr is an involution in X„ with vri = t > n + 2 — k, then the entry t does 
not appear in any occurrence of r G Ak in vr. Thus, there exists a bijection between the set 
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of involutions vr G In{Ak) with vri = t>n + 2 — k and the set of involutions vr' G T„_2(^fc)- 
Therefore gfc(n; t) = gk(n — 2), as claimed. □ 

Let Gk{n]v) be the polynomial X]"=i S'fcl'^; Rewriting the above lemma in terms of 

the polynomials Gk{n; v) we have the following recurrence relation. 

Lemma 3.3. Let k >3. For all n>k, 

Gkin;v) 

= fk-i{n - 1) + vfk-i{n - 2) - v{k - 2)fk-i{n - 3) + + (A; - 2)v^ Gk{n - 2; v) 
-^,Gk{n - 2; 1) + (fc - 2 + --^) G,{n - 4; 1), 



where Gk{n; v) = In-i + In-2 for all n = 0,1, ... ,k — 1. 



Proof. Lemma |3.2| gives 
Gk{n;v) 



n—k I t—\ 



fk-i{n - 1) + vfk^iin -2) + Y,v' \{k- 2)Gk{n - 2; t) + ^ Gk{n - 2; j) 

t=2 \ j=l 



n-1 

,t 



+Gk{n -2;1) J2 

t=n+l-k 

(n-3 
Gk{n-2;v)-Gk{n-4;l) 

-"^{Gkin -2;l)-{k- l)G,(n - 4; 1)) + -"^[l'--" Gk{n - 2; 1) 
Gfc(n-2;t;)-A_i(n-3)-Gfc(n-4;l) 

j=n-k 

which is equivalent to 
Gk{n;v) 

= /fc_i(n - 1) + vfk-i{n - 2) - v{k - 2)fk-i{n - 3) + ( + (A; - 2)t;) - 2; v) 
-f^^Gk{n - 2; 1) + ^ (fc - 2 + - 4; 1). 

To find the value of Gk{n;v) for n < — 1, let vr be any involution with tti = t. If t = 1 
then there are In-i involutions, whereas if t > 1 there are In-2 involutions, hence Gk{n; v) = 
v^In-i + I]"=2 v^~^In-2 = In-1 + ^j^In-2, as required. □ 



Lemma 3.3 can be generalised as follows; let gk;m{n;t) be the number of involutions vr G 
Ini-^k) such that TTi = t and vr contains exactly m fixed points. Define Gk{n;t;p) = 
Ylm=o 9>^;rn{n;t)p'^ and Gk{n;v,p) = Yl^=iGk{n;t;p)v*~^. Using the same arguments as 
those in the proofs of Lemma |3.2| and Lemma |3.3[ while carefully considering the number of 
fixed points, we have the following result. 
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Lemma 3.4. Let k >3. For all n > k, 

Gk{n;v,p) 

= pfk-i{n - 1) + vfk-i{n - 2) - pv{k - 2)fk-i{n - 3) + + (A; - 2)^;) Gk{n - 2; v,p) 
-i^Gkin - 2; l,p) + (fc - 2 + --^) Gk{n - 4; 

where Gk{n; v,p) = pJn-i{p) + ^jir^Jn-2{p) for all n = 0,1, ... ,k - 1. 



Let Gk{x, v,p) = Ylin>Q Gk{n; v,p)x^ be the generating function for the sequence Gk{n; v,p). 
Define Ji{v,p) to be the polynomial Y2 dtrV^P^ where dtr is the number of involutions tt G li 
such that TTi = t + 1 and vr contains exactly r fixed points. Rewriting the recurrence relation 
in the statement of Lemma 3.4 in terms of generating functions we obtain 

Gk{x,v,p) = 

k-l ( fe-2 \ / fc-3 

E Jjiv,p)x3 j^px Fk-i{x,p) - Yl Jj{p)x^ \+vx'^[ Fk-i{x,p) - Yl Jj{p)x^ 

j=0 \ j=0 J \ j=0 

-{k - 2)pvx' (Fk-i{x,p) - YMp)^'] - \Gk{xv, l,p) - YMp)i^^y 
W + A; - 2) (Gk{x,v,p) - Y^M^,P)x'^ 

^ijszM^ (Gk{xv,i,p)-YMp){xvy] - (Gk{xv,i,p)-YMp){xvy 



yk-6ri_^\2 

j=0 J \ j=o 



which is equivalent to 



l-v 

r.2 



[k-2)^)Gk[x/v,v,p) 



+ 'i:JA^.P)1j + '-^{Fk-i{x/v,p)-YJMfA+'v\Fk^^^^ 
-{k - 2)p-^ [Fk-,{x/v,p) - YJ,{p)fA + 1^ YJj{p)x^ 

^ ' j=0 3=0 ^ ' j=0 

To solve this functional equation, we substitute 

1 



V ■-yQ = -yl + [k- 3)x2 + - 2{k - l)j;2 + {k- 3)2x4 j , 

where vq is the root of the coefficient of Gk{x/v, v,p) above, into the above functional equa- 
tion, that is, 1 - ^ - (A; - 2)g = 0. Since J,{v,p) = pJ,-i{p) + ^Jj-2{p) for ah 
j = 1, 2, . . . , A; — 1 and Jo{v,p) = 1, it is routine to show (via some rather tedious algebraic 
manipulation) that we obtain Theorem 
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